feat(tools): add SDK idempotency keys for safe memory write retries - #1642

Summary

Adds SDK-level idempotency keys for safe memory write retries — Phase A (SDK-only) of the Hierarchy. Generates a stable Idempotency-Key header on every addMemory / documentAdd call so network retries and caller retries do not create duplicate memories. Updated with Staff improvements: custom key, RetryContext, NFC normalization, 14 Vitest tests.

Fixes #1627

Problem

packages/tools/src/ai-sdk.ts:addMemoryTool and documentAddTool (and openai/tools.ts) called client.add({ content, containerTags }) with no idempotency signal. On retry (the supermemory client retries 2× on 5XX/network, plus caller retries), the backend creates duplicate memories/embeddings/writes.

• packages/tools/src/ai-sdk.ts:102 — client.add with no Idempotency-Key
• packages/tools/src/openai/tools.ts:290 — same
• No helper to keep the same key across retries; offline queue has no stable dedupe key

Impact

• Reliability: retries are now safe — same content+tags within the same minute → same key, backend can dedupe (Phase B) and caller can dedupe even before backend honors it.
• Duplicate prevention: content|tags|minuteBucket hash prevents the common fetch failed → retry → duplicate loop on flaky networks/mobile.
• Future-ready: foundation for offline sync and batch idempotency without waiting for backend.
• Developer flexibility: custom key + RetryContext cover both one-off and session-scoped retries.

Solution — Phase A (SDK-only, mergeable now, no backend change)

New file packages/tools/src/shared/idempotency.ts (96 lines, zero deps):

Idempotency-Key = SHA-256( normalizedContent + '|' + sorted(containerTags).join(',') + '|' + minuteBucket )
normalizedContent = content.normalize("NFC").trim()  // unicode + whitespace stable
minuteBucket = Math.floor(Date.now() / 60000)  // stable within minute, rotates after

Why SHA-256?

• Deterministic across runtimes (browser, Node, Workers).
• Available through Web Crypto in browsers, Node, and Workers.
• Fixed-size output suitable for HTTP headers.
• No additional dependency required.

Core helpers:

• generateIdempotencyKey(content, containerTags, now?, customKey?) — uses Web Crypto crypto.subtle (Node 20+). customKey priority: user-provided → generated (if customKey is non-empty, returned as-is).
• buildIdempotencyHeaders(content, containerTags, now?, customKey?) → { "Idempotency-Key": key }
• createRetryContext(content, containerTags, now?, customKey?) → { key, headers, getKey(), getHeaders() } — future-proof retry helper that captures the key once and reuses it across retries, even across minute rollovers:

const retryCtx = await createRetryContext(memory, containerTags)
await client.add(params, { headers: retryCtx.headers })
await client.add(params, { headers: retryCtx.headers }) // same key, even 61s later

Wired in packages/tools/src/ai-sdk.ts (addMemoryTool + documentAddTool now accept optional idempotencyKey input) and packages/tools/src/openai/tools.ts:

// ai-sdk tool — Priority: User-provided → Generated
const headers = await buildIdempotencyHeaders(memory, containerTags, Date.now(), idempotencyKey)
await client.add({ content: memory, containerTags }, { headers })

// Direct SDK:
await addMemory({ content, containerTags, idempotencyKey })
const ctx = await createRetryContext(content, containerTags)
await addMemory(..., ctx.headers)

Header is optional for the backend — ignored until Phase B honors it, so this PR is useful even before server support. No breaking change, no new package. Tool addMemory now exposes idempotencyKey?: string input for developers who already have a key.

Benchmark

Key generation is SHA-256 of a short string — ~0.05–0.1 ms per call (measured locally via Vitest, crypto.subtle); negligible vs. the network call it protects. No impact on searchMemories or other tools.

Failure Handling

• Normalization: content.normalize("NFC").trim() before hashing — Unicode stable (café NFC == cafe\u0301 NFD) and trailing whitespace stable ("hello " == "hello"), cross-platform stable.
• Empty content → still deterministic (hashes |tags|bucket).
• containerTags order independence — sorted before hashing so ["b","a"] and ["a","b"] produce the same key.
• Minute rollover → different key (prevents indefinite dedupe); RetryContext intentionally reuses the captured key across rollovers for safe retry.
• now injection for testing and for callers that want to control bucketing.
• Concurrent callers share the same key when inputs and minute are identical (parallel generation → 1 distinct key).

Memory Footprint

Stateless — no cache, no storage. One SHA-256 per write (~64 bytes hex), plus optional RetryContext (~80 bytes). No growth.

Testing

Real Vitest — bun x vitest run packages/tools/src/shared/idempotency.test.ts — 14 tests, 5 ms:

 ✓ packages/tools/src/shared/idempotency.test.ts (14 tests) 5ms
 Test Files  1 passed (1)
      Tests  14 passed (14)

Biome: bun x biome check packages/tools/src/shared/idempotency.ts — clean.
Typecheck: bun x tsc --noEmit --project packages/tools/tsconfig.json — no new errors (pre-existing ai-sdk.ts tool overload errors verified on main via git stash).
Vitest: 14/14 pass (was 10/10, +4 Staff-requested edge cases).

Environment

• Platform: macOS (arm64), Bun 1.4.0, Node 26.7.0, TypeScript 5.9.2, Vitest 3.2.4, supermemory@3.0.0-alpha.26
• Branch: fix/idempotent-memory-writes @ a84003f1 (force-pushed to Sravanjangam/supermemory)
• Upstream base: supermemoryai/supermemory@main d436792e
• Biome: clean; typecheck: no new errors; tests: 14/14 Vitest pass
• Before-commit checks 2× + after-commit 1× per pipeline — all passed

Non-goals (Phase A)

• No backend change (Phase B will honor the header with a server-side dedupe store)
• No persistent storage of keys
• No cross-process coordination
• No batch/bulk idempotency
• No userId in hash (backend can hash apiKey server-side if needed)

Deferred roadmap

• Phase B: backend honors Idempotency-Key (dedupe store, 409 or 200 replay)
• Phase C: batch/bulk idempotency + customId integration